Distinction of some induced representations

Let K/F be a quadratic extension of p-adic fields, σ the nontrivial element of the Galois group of K over F , and ∆ a quasi-square-integrable representation of GL(n,K). Denoting by ∆∨ the smooth contragredient of ∆, and by ∆σ the representation ∆ ◦ σ, we show that representation ofGL(2n,K) obtained by normalized parabolic induction of the representation ∆∨⊗∆σ, is distinguished with respect to GL(2n, F ). This is a step towards the classification of distinguished generic representations of general linear groups over p-adic fields. Introduction Let K/F be a quadratic extension of p-adic fields, σ the nontrivial element of the Galois group of K over F , and ∆ a quasi-square-integrable representation of GL(n,K). We denote by σ again the automorphism of M2n(K) induced by σ. If χ is a character of F , a smooth representation ρ of GL(2n,K) is said to be χ-distinguished if there is a nonzero linear form L on its space V , verifying L(ρ(h)v) = χ(det(h))L(v) for all h in GL(2n, F ) and v in V , we say distinguished if χ = 1. If ρ is irreducible, the space of such linear forms is of dimension at most 1 (Proposition 11 of [F2]). Calling ∆ the smooth contragredient of ∆ and ∆ the representation ∆◦σ, we denote by ∆×∆ the representation of GL(2n,K), obtained by normalized induction of the representation ∆⊗∆ of the standard parabolic subgroup of type (n, n). The aim of the present work is to show that the representation ∆ ×∆ is distinguished. The case n = 1 is treated in [H] for unitary ∆×∆, using a criterion characterizing distinction in terms of gamma factors. In [F3], Flicker defines a linear form on the space of ∆×∆ by a formal integral which would define the invariant linear form once the convergence is insured. Finally in [F-H], for n = 1, the convergence of this linear form is obtained for ∆| |K × ∆ | | K and s of real part large enough when ∆ is unitary, the conclusion follows from an analytic continuation argument. We generalize this method here. The first section is about notations and basic concepts used in the rest of the work. In the second section, we state a theorem of Bernstein (Theorem 2.1) about rationality of solutions of polynomial systems, and use it as in [C-P] or [Ba], in order to show, in Proposition 2.2, the holomorphy of integrals of Whittaker functions depending on several complex variables. The third section is devoted to the proof of theorem 3.1, which asserts that the representation ∆| |K ×∆ | | K is distinguished when ∆ is unitary and Re(s) is in a neighbourhood of n. In the fourth section, we extend the result in Theorem 4.2 to every complex number s. Our proof relies decisively on a theorem of Youngbin Ok (Proposition 2.3 of the present paper), which is a twisted version of a well-known theorem of Bernstein ([Ber], Theorem A). We end this introduction by recalling a conjecture about classification of distinguished generic representations: Conjecture. Let m be a positive integer, and ρ a generic representation of the group GL(m,K), obtained by normalized parabolic induction of quasi-square-integrable representations ∆1, . . . ,∆t. It is distinguished if and only if there exists a reordering of the ∆i’s, and an integer r between 1 and t/2, such that we have ∆i+1 = ∆ ∨ i for i = 1, 3, .., 2r− 1, and ∆i is distinguished for i > 2r. 1 ha l-0 03 40 84 3, v er si on 1 3 2 1 O ct 2 00 9 We denote by η the nontrivial character of F ∗ trivial on the norms of K. According to Proposition 26 in [F1], Proposition 12 of [F2], Theorem 6 of [K], and Corollary 1.6 [A-K-T], our result reduces the proof of the conjecture to show that representations of the form ∆1 × · · · ×∆t with ∆i+1 = ∆ ∨ i for i = 1, 3, .., 2r − 1 for some r between 1 and t/2, and non isomorphic distinguished or η-distinguished ∆i’s for i > 2r are not distinguished whenever one of the ∆i’s is η-distinguished for i > 2r. According to [M3], the preceding conjecture implies the equality of the analytically defined Asai L-function and the Galois Asai L-function of a generic representation. 1 Notations We denote by | |K and | |F the respective absolute values on K , by qK and qF the respective cardinalities of their residual field, by RK the valuation ring of K, and by PK the maximal ideal of RK . The restriction of | |K to F is equal to | | 2 F . More generally, if the context is clear, we denote by |M |K and |M |F the positive numbers |det(M)|K and |det(M)|F for M a square matrix with determinant in K and F respectively. We denote by Gn the algebraic group GL(n). Hence if π is a representation of Gn(K) for some positive n, and if s is a complex number, we denote by π| |K the twist of π by the character |det( )|K . We call partition of a positive integer n, a family n̄ = (n1, . . . , nt) of positive integers (for a certain t in N − {0}), such that the sum n1 + · · · + nt is equal to n. To such a partition, we associate a subgroup of Gn(K) denoted by Pn̄(K), given by matrices of the form